1 Perceptron From Scratch

The Perceptron

Today we will build a perceptron that will allow us to determine whether a patient has a sleep disorder or not. We will build it and train it from scratch using only the numpy library, with the additional support of pandas to load the data and matplotlib to visualize the training process and results.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

Exercise 1 - Load the data from the sleep_disorder_data.csv file with pandas and visualize it in the notebook.

data = pd.read_csv('sleep_disorder_data.csv', index_col=False)
data

Exercise 2 - Select the columns ‘Quality of Sleep’, ‘Physical Activity Level’ and ‘Stress Level’ as your input data X and the column ‘Sleep Disorder’ as your target y and convert them to numpy arrays. Print the labels stored in y.

Exercise 3 - You should see that you have three different labels: ‘Sleep Apnea’, ‘Insomnia’ and ‘None’. The first two correspond to sleeping disorders, while the label ‘None’ refers to the absence of sleep disorder (normal sleep). Convert the labels to numerical values such that each ‘None’ becomes 0 and ‘Sleep Apnea’ and ‘Insomnia’ both become 1. Convert your numpy array to integer data type.

data.replace(np.nan, 0, inplace=True)
data.replace('Sleep Apnea', 1, inplace=True)
data.replace('Insomnia', 1, inplace=True)
X = data[['Quality of Sleep', 'Physical Activity Level', 'Stress Level']].to_numpy()
y = data['Sleep Disorder'].to_numpy().astype(int)

Exercise 4 - Standardize the data matrix X such that each element in the resulting matrix is the following:

\[\begin{equation} x_i^j:=\frac{x_i^j-\mu^j}{\sigma^j}, \end{equation}\]

where \(x_i^j\) is the \(i\)th data point in the \(j\)th column and \(\mu_j\) and \(\sigma_j\) are the mean and standard deviation, respectively, of the \(j\)th column (note that \(\sigma_j\) refers to the standard deviation of column \(j\) of the feature space and is not related to the sigmoid function \(\sigma\))

X_standard = (X - np.mean(X, axis=0)) / np.std(X, axis=0)

Now it is time to start building the model. Since the elements in the training set are 3 dimensional (we are looking at 3 features: ‘Quality of Sleep’, ‘Physical Activity Level’ and ‘Stress Level’) our single neuron model will have 3 weights and 1 bias.

Exercise 5 - Initialize a matrix of random weights of shape (3,1) and a bias of shape (1,) between -1 and 1, store them in two variables called weights and bias and print them. Use the random seed 6 for reproducibility.

np.random.seed(6)
weights = np.random.random((3,1)) * 2 - 1
bias = np.random.rand(1)

print('weights:', W)
print('bias', b)

The next step is to define the forward function. The output is given by the weighted sum of the inputs plus the bias, which then goes through a sigmoid function to produce the final prediction \(\hat{y}\):

\[\begin{equation} \hat{y}=\sigma(w_1x_1+w_2x_2+w_3x_3+b) \end{equation}\]

Given an input column vector \(\vec{x}=\begin{bmatrix}x_1 \\ x_2 \\ x_3\end{bmatrix}\), a weight matrix \(W=\begin{bmatrix}w_1 \\ w_2 \\ w_3\end{bmatrix}\) and a bias \(b\) we can express the linear combination of the weights and inputs plus the bias in matrix form as

\[\begin{equation} W^T\vec{x}+b=\begin{bmatrix}w_1 & w_2 & w_3\end{bmatrix}\begin{bmatrix}x_1 \\ x_2 \\ x_3\end{bmatrix}+b=w_1x_1+w_2x_2+w_3x_3+b \end{equation}\]

which gives the following expression for the prediction:

\[\begin{equation} \hat{y}=\sigma(W^T\vec{x}+b) \end{equation}\]

In our case, since each column represents a feature and observations are represented by rows, we also need to transpose the input matrix X.

Exercise 6 - Define a function called sigmoid that takes an input x and returns the following:

\[\begin{equation} \sigma(x)=\frac{1}{1+e^{-x}} \end{equation}\]

remember that you can use np.exp(x) to raise \(e^x\)

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

Exercise 7 - Define a function called forward that takes 3 inputs: the weights, the bias and the data matrix X and returns the output of the perceptron:

\[\begin{equation} \hat{y}=\sigma(W^TX+b) \end{equation}\]

Tip: use the previously defined sigmoid function.

def forward(w, b, X):
    return sigmoid(np.dot(X, w) + b)

Exercise 8 - Make an initial prediction with the untrained model using the forward function that you just defined. You should visualize two outputs: the plain output and the rounded output, which will be 1 or 0 for each data point. Compare your predictions with the target \(y\). How many did you get right?

y_raw = forward(W, b, X_standard)
y_pred = (y_raw > 0.5).astype(int).flatten()
accuracy = (y == y_pred).sum() / len(y)

Exercise 9 - Define a function called mse_loss that calculates the mean squared error:

\[\begin{equation} MSE=\frac{1}{n}\sum(y-\hat{y})^2 \end{equation}\]

Before moving on to the final training loop, we need to define a function that updates the weights according to the value of the gradient, which gives us information about which direction we have to move in order to minimize the loss function. The derivative of the sigmoid function is \(\sigma(z)'=\sigma(z)(1-\sigma(z))\) and the gradient of the loss function \(L\) for some input vector \(\vec{x}\) and target \(y\) is given by

\[\begin{equation} \nabla L(x,y)=\begin{bmatrix} \frac{\partial L}{\partial w_1} \\ \frac{\partial L}{\partial w_2} \\ \frac{\partial L}{\partial w_3} \\ \frac{\partial L}{\partial b} \end{bmatrix}=\begin{bmatrix} -2x_1\left[\left(1-\sigma(x)\right)\sigma(x)\right]\left[y-\sigma(x)\right] \\ -2x_2\left[\left(1-\sigma(x)\right)\sigma(x)\right]\left[y-\sigma(x)\right] \\ -2x_3\left[\left(1-\sigma(x)\right)\sigma(x)\right]\left[y-\sigma(x)\right] \\ -2\left[\left(1-\sigma(x)\right)\sigma(x)\right]\left[y-\sigma(x)\right] \end{bmatrix} \end{equation}\]

def loss(y, y_pred):
    return np.mean((y - y_pred)**2)

initial_loss = loss(y, y_raw)
print(initial_loss)

Exercise 10 - Define a function called compute_gradient that takes as input the data matrix \(X\) the target \(y\), the weights and the bias and returns the gradient of the loss function \(L\) as a tuple: the first value is an array with three entries consisting of the derivative of the loss function with respect to each weight and the second value is the derivative of the loss function with respect to the bias.

def forward(data, weights, bias):
    return sigmoid(np.dot(weights.T, data.T) + bias)

y_pred = forward(X_standard, W, b)

def compute_gradient(X, y, w, b):
    y_pred = forward(X, w, b).flatten()
    grad = 2 * (y - y_pred) * ((1 - y_pred) * y_pred)
    b_grad = np.mean(grad)
    w_grad = np.mean(X * grad.reshape(-1,1), axis=0)
    return w_grad, b_grad

w_grad, b_grad = compute_gradient(X_standard, y, W, b)

Finally, we need to define the training loop. At each step, we will use X as the input and y as the target that we will compare to the output, and we will update the weights such that the error decreases.

Exercise 11 - Define a training loop with 30 epochs and learning rate \(\alpha=0.1\). Use the previously defined compute_gradient function and update the weights accordingly. At each step, print the loss and store it in a list.

W = np.random.rand(3).reshape(3, 1) * 2 - 1
b = np.random.rand(1)

y_raw = forward(X_standard, W, b).flatten()
y_pred = (y_raw > 0.5).astype(int)
accuracy = 100 * np.sum(y_pred == y).item() / len(y)
print(f'Initial accuracy is {round(accuracy, 2)}%')

epochs = 500
alpha = 0.25

loss_per_epoch = []
for epoch in range(epochs):
    y_pred = forward(X_standard, W, b).flatten()
    loss_epoch = loss(y, y_pred)
    loss_per_epoch.append(loss_epoch)
    w_grad, b_grad = compute_gradient(X_standard, y, W, b)
    W += alpha * w_grad.reshape(-1,1)
    b += alpha * b_grad

y_raw = forward(X_standard, W, b).flatten()
y_pred = (y_raw > 0.5).astype(int)
accuracy = 100 * np.sum(y_pred == y).item() / len(y)
print(f'final accuracy is {round(accuracy, 2)}%')

Exercise 12 - Plot the loss per epoch. Label the x-axis ‘Epoch’ and the y-axis ‘Loss (MSE)’, both with font size 14. Is the perceptron learning from the training data?

plt.plot(loss_per_epoch)

Exercise 13 - Print the final weights matrix and the bias. How do they compare with the initial weights and bias from exercise 4? Which features are positively correlated and which ones negatively correlated with the presence of a sleep disorder?

Exercise 14 - Make a final prediction with the trained model and compare it to the target \(y\). How many of your predictions are correct? What is the accuracy of your model?

y_raw = forward(X_standard, W, b).flatten()
y_pred = (y_raw > 0.5).astype(int)
accuracy = 100 * np.sum(y_pred == y).item() / len(y)
print(f'accuracy is {round(accuracy, 2)}%')

Exercise: write your answer in this editable Python cell.